Rust memory engine: 6-layer parallel recall with full ORM integration#260
Rust memory engine: 6-layer parallel recall with full ORM integration#260
Conversation
Rust engine (4 new files, 1839 lines): - channel_types.rs: ActivityDomain enum, QueueItemBehavior trait with RTOS aging - channel_items.rs: Voice (always urgent, never kicked), Chat (per-room consolidation), Task (dependency-aware, overdue urgency) + ChannelEnqueueRequest discriminated union - channel_queue.rs: Generic priority queue with capacity kick management - channel_registry.rs: Domain routing, service_cycle() with state-gated scheduling IPC layer (5 new commands): - channel/enqueue, channel/dequeue, channel/status, channel/service-cycle, channel/clear - Per-persona registry auto-creation with Arc<Mutex<HashMap>> TypeScript bridge: - RustCoreIPC.ts: 5 low-level IPC methods - RustCognitionBridge.ts: 4 high-level bridge methods with forensic logging - PersonaInbox.ts: Rust-first routing with TS fallback - QueueItemTypes.ts: toChannelEnqueueRequest() factory - 6 ts-rs generated type exports 232 tests pass, zero clippy warnings in new code.
CNS service cycle now asks Rust "what should I process next?" via IPC: - serviceViaRust(): calls rustBridge.serviceCycle() for <1ms scheduling - Rust handles consolidation, state gating, urgent-first ordering - TS just executes what Rust decided (LLM calls, DB, tools) - Falls back to TS channels if Rust bridge unavailable Wiring: - CNSTypes.ts: added optional rustBridge to CNSConfig - CNSFactory.ts: passes rustCognitionBridge from PersonaUser - PersonaUser.ts: added nullable rustCognitionBridge getter, wired inbox.setRustBridge() after bridge initialization - QueueItemTypes.ts: added fromRustServiceItem() converter (Rust camelCase JSON → TS QueueItem discriminated union) 232 Rust tests pass, TypeScript builds clean.
… forwarding - Delete 9 dead TS files: channel queue system (6 files) and cognitive schedulers (3 files) now fully replaced by Rust cognition engine - Slim down CNS, PersonaInbox, PersonaAutonomousLoop, PersonaUser to remove all fallback/legacy TS scheduling paths - Fix @reactive() and @attr() decorators: TC39 class fields shadow Lit's prototype accessor from createProperty(), preventing re-renders. queueMicrotask removes the own property after init so Lit's setter triggers requestUpdate() automatically. - Forward all transcription captions to all participants (including speaker's own) — captions confirm system is working, audio echo is handled by mix-minus in the Rust mixer, not transcription filtering.
Phase A: Built complete memory system in Rust (corpus, recall, consciousness) - MemoryCorpus: per-persona in-memory data (memories + embeddings + timeline events) - 6-layer multi-recall: Core, Semantic, Temporal, Associative, DecayResurface, CrossContext - Consciousness context: temporal awareness + cross-context peripheral awareness - DashMap<Arc<MemoryCorpus>> concurrency: lock-free reads, COW writes - Deterministic embeddings for offline recall (no external model dependency) - 293 tests passing (277 lib + 16 recall accuracy integration tests) Phase B: Wired Rust ↔ TypeScript via IPC - Cleaned dead TS methods (memoryInit/Store/Recall/RecordEvent/Consolidate) - Added memoryLoadCorpus: bulk ORM → Rust at startup - Added memoryAppendMemory/Event: incremental COW updates on DB writes - Hippocampus: appends to Rust corpus after every DataCreate to longterm.db - UnifiedConsciousness: appends timeline events to Rust corpus after recording - PersonaUser: loads full corpus from ORM at startup, wires bridge to consciousness - SemanticMemorySource + GlobalAwarenessSource: call Rust recall (sub-ms response)
There was a problem hiding this comment.
Pull request overview
This PR introduces a complete Rust-based memory system with 6-layer parallel recall and multi-channel queue management. The implementation migrates critical memory operations from TypeScript to Rust for performance improvements (3-60s → ~30ms for memory operations).
Changes:
- Built complete Rust memory subsystem: MemoryCorpus, 6-layer recall (Core, Semantic, Temporal, Associative, DecayResurface, CrossContext), consciousness context builder, timeline queries
- Integrated Rust ↔ TypeScript: bulk corpus loading from ORM at startup, incremental COW append on DB writes, DashMap<Arc> concurrency model
- Replaced TS-based channel system with Rust multi-domain queue: AUDIO/CHAT/BACKGROUND channels with item polymorphism, RTOS aging, consolidation
Reviewed changes
Copilot reviewed 67 out of 76 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| continuum-core/src/memory/*.rs | New Rust memory engine with types, corpus, embedding, consciousness, timeline, and cache modules |
| continuum-core/src/persona/channel_*.rs | New Rust channel system replacing TS BaseQueueItem/ChannelQueue/ChannelRegistry |
| continuum-core/src/main.rs | Initialize FastEmbed embedding provider and PersonaMemoryManager |
| continuum-core/tests/common/mod.rs | Shared IPC client utilities with length-prefixed binary framing protocol |
| bindings/*.ts | Generated TypeScript types for Rust memory and channel systems |
| RustCognitionBridge.ts | Added memory and channel system IPC methods |
| PersonaUser.ts | Integrated corpus loading from ORM and Rust bridge wiring |
| PersonaCentralNervousSystem.ts | Replaced TS scheduling with Rust-delegated service_cycle |
| CNSFactory.ts | Simplified to create Rust-delegated CNS instances |
| QueueItemTypes.ts | Added conversions between TS queue items and Rust channel enqueue requests |
| PersonaInbox.ts | Routes items through Rust channel system via IPC |
| SemanticMemorySource.ts | Delegates to Rust multi-layer recall via IPC |
| GlobalAwarenessSource.ts | Delegates to Rust consciousness context builder via IPC |
| Deleted files | Removed TS channel system (BaseQueueItem.ts, ChannelQueue.ts, etc.) and cognitive schedulers (ICognitiveScheduler.ts, HeuristicCognitiveScheduler.ts, etc.) |
Files not reviewed (1)
- src/debug/jtag/package-lock.json: Language not supported
Comments suppressed due to low confidence (17)
src/debug/jtag/system/user/server/PersonaUser.ts:23
- Unused import Commands.
import { Commands } from '../../core/shared/Commands';
src/debug/jtag/system/user/server/PersonaUser.ts:38
- Unused import TaskEntity.
import { TaskEntity } from '../../data/entities/TaskEntity';
src/debug/jtag/system/user/server/PersonaUser.ts:39
- Unused import taskEntityToInboxTask.
import { taskEntityToInboxTask } from './modules/QueueItemTypes';
src/debug/jtag/system/user/server/PersonaUser.ts:42
- Unused import ChatRAGBuilder.
import { ChatRAGBuilder } from '../../rag/builders/ChatRAGBuilder';
src/debug/jtag/system/user/server/PersonaUser.ts:47
- Unused import AIDecisionService.
import { AIDecisionService, type AIDecisionContext } from '../../ai/server/AIDecisionService';
src/debug/jtag/system/user/server/PersonaUser.ts:49
- Unused import CoordinationDecisionLogger.
import { CoordinationDecisionLogger, type LogDecisionParams } from '../../coordination/server/CoordinationDecisionLogger';
src/debug/jtag/system/user/server/PersonaUser.ts:61
- Unused import AI_DECISION_EVENTS.
import {
AI_DECISION_EVENTS,
type AIEvaluatingEventData,
type AIDecidedRespondEventData,
type AIDecidedSilentEventData,
type AIGeneratingEventData,
type AICheckingRedundancyEventData,
type AIPostedEventData,
type AIErrorEventData
} from '../../events/shared/AIDecisionEvents';
src/debug/jtag/system/user/server/PersonaUser.ts:68
- Unused import AI_LEARNING_EVENTS.
import {
AI_LEARNING_EVENTS,
type AITrainingStartedEventData,
type AITrainingCompleteEventData,
type AITrainingErrorEventData,
type AIInteractionCapturedEventData
} from '../../events/shared/AILearningEvents';
src/debug/jtag/system/user/server/PersonaUser.ts:82
- Unused import PersonaGenome.
import { PersonaGenome, type PersonaGenomeConfig } from './modules/PersonaGenome';
src/debug/jtag/system/user/server/PersonaUser.ts:95
- Unused import CognitionLogger.
import { CognitionLogger } from './modules/cognition/CognitionLogger';
src/debug/jtag/system/user/server/modules/PersonaMessageEvaluator.ts:31
- Unused import AIDecisionService.
import { AIDecisionService } from '../../../ai/server/AIDecisionService';
src/debug/jtag/system/user/server/modules/PersonaMessageEvaluator.ts:53
- Unused import getStageStatus.
import {
COGNITION_EVENTS,
type StageCompleteEvent,
calculateSpeedScore,
getStageStatus
} from '../../../conversation/shared/CognitionEventTypes';
src/debug/jtag/system/user/server/modules/cognitive/memory/Hippocampus.ts:18
- Unused import DATA_COMMANDS.
src/debug/jtag/system/user/server/modules/cognitive/memory/Hippocampus.ts:20 - Unused import Commands.
src/debug/jtag/system/user/server/modules/cognitive/memory/Hippocampus.ts:27 - Unused import generateUUID.
src/debug/jtag/system/user/server/modules/cognitive/memory/Hippocampus.ts:39 - Unused import RawMemoryAdapter.
src/debug/jtag/system/user/server/modules/cognitive/memory/Hippocampus.ts:41 - Unused import DataDaemon.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| @@ -69,7 +69,7 @@ import { | |||
| import { Events } from '../../core/shared/Events'; | |||
| import { EVENT_SCOPES } from '../../events/shared/EventSystemConstants'; | |||
There was a problem hiding this comment.
Unused import EVENT_SCOPES.
| import { EVENT_SCOPES } from '../../events/shared/EventSystemConstants'; |
| import type { DataListParams, DataListResult } from '../../../commands/data/list/shared/DataListTypes'; | ||
| import { DataList, type DataListParams, type DataListResult } from '../../../commands/data/list/shared/DataListTypes'; | ||
| import type { StageCompleteEvent } from '../../conversation/shared/CognitionEventTypes'; | ||
| import { calculateSpeedScore, getStageStatus, COGNITION_EVENTS } from '../../conversation/shared/CognitionEventTypes'; |
There was a problem hiding this comment.
Unused imports COGNITION_EVENTS, calculateSpeedScore, getStageStatus.
| import { calculateSpeedScore, getStageStatus, COGNITION_EVENTS } from '../../conversation/shared/CognitionEventTypes'; |
Summary
Test plan
cargo test --release -p continuum-core— 277 lib tests passcargo test --release -p continuum-core --test memory_recall_accuracy— 16 recall accuracy tests passnpm run build:ts— TypeScript compilation succeedsnpm startand verify persona logs show corpus loading